Detect property to exclude Cargo dependencies #1421

zahidblackduck · 2025-05-06T13:42:42Z

JIRA Ticket

IDETECT-4326

Description

This merge request introduces a feature to exclude specific types of dependencies in the Cargo detector, providing more control over which dependencies are included in detect scans. The supported types are:

Normal: Regular dependencies needed for runtime.
Dev: Development dependencies, such as test libraries.
Build: Build-related dependencies.
Proc-Macro: Procedural macro dependencies.

A new property has been added to configure the exclusion of these dependencies:

--detect.cargo.dependency.types.excluded=NONE,NORMAL,DEV,BUILD,PROC_MACRO

By default, all dependencies are included (NONE). Specific types can be excluded by specifying the appropriate values.

Implementation Details

To support this feature, an enum (CargoDependencyType) has been introduced to represent the dependency types. This logic has been implemented across both Cargo CLI Detector and Cargo Lockfile Detector:

Cargo CLI Detector: Uses the cargo tree command with native options to exclude dependencies. For example, excluding development dependencies is done by adding the --edges no-dev flag to the command. This solution leverages the CLI's built-in functionality with minimal post-processing.
Cargo Lockfile Detector: Since Cargo.lock does not include dependency types, the lockfile detector reads and parses the Cargo.toml file. It collects the [dev-dependencies] and [build-dependencies] sections, filtering out matching dependencies based on the exclusion configuration.

.../src/main/java/com/blackduck/integration/detectable/detectables/cargo/CargoCliExtractor.java

.../main/java/com/blackduck/integration/detectable/detectables/cargo/parse/CargoTomlParser.java

andrian-sevastyanov · 2025-05-06T14:48:59Z

...ble/src/main/java/com/blackduck/integration/detectable/detectables/cargo/CargoExtractor.java

-import java.util.ArrayList;
-import java.util.List;
-import java.util.Optional;
+import java.util.*;


Another wildcard.

...ble/src/main/java/com/blackduck/integration/detectable/detectables/cargo/CargoExtractor.java

devmehtabd · 2025-05-07T19:17:51Z

...ble/src/main/java/com/blackduck/integration/detectable/detectables/cargo/CargoExtractor.java

+        List<CargoLockPackageData> filteredPackages = cargoLockPackageDataList;
+        String cargoTomlContents = FileUtils.readFileToString(cargoTomlFile, StandardCharsets.UTF_8);
+
+        if (cargoDetectableOptions != null) {


Instead of checking on cargoDetectableOptions object, can we check on dependencyTypeFilter instead? There could be more properties added to this detector in future not related to dependency exclusion.

Ah, you're right. The actual filtering is handled by dependencyTypeFilter. This null check is just to verify whether any options were passed at all. The filtering logic using dependencyTypeFilter is applied later in the CargoTomlParser#parseDependenciesToExclude(..) method.

I understand that, my only concern is that if in future lets say we add some new property for excluding workspaces as an example then those options will also be passed in CargoDetectableOptions, so we will make this unnecessary call to cargoTomlParser which is not required if no dependency exclusion is involved. I hope I made sense.

The current code setup doesn't call the methods of CargoTomlParser if no dependency exclusion is involved. Nonetheless, I aim to address the issue that you pointed out in any future enhancement of cargo related issue.

I know, currently your code is not getting executed if cargoDetectableOptions is null. It is not guaranteed that you are going to work on this in any future enhancement related to Cargo and it could be easily missed. I believe we should address this now, its a simple change and not a complicated ask.

Okay, sure. I'll update accordingly.

devmehtabd · 2025-05-07T19:20:14Z

src/main/java/com/blackduck/integration/detect/configuration/DetectProperties.java

+            .setInfo("Cargo Dependency Types Excluded", DetectPropertyFromVersion.VERSION_10_5_0)
+            .setHelp(
+                "A comma-separated list of dependency types that will be excluded.",
+                "If DEV is excluded, the Cargo CLI Detector will exclude 'dev' dependencies when parsing the output of cargo tree."


Should be changed, since this is applicable to Lockfile Detector as well.

It's a good find. I'll update.

devmehtabd · 2025-05-07T19:22:26Z

.../main/java/com/blackduck/integration/detectable/detectables/cargo/parse/CargoTomlParser.java

@@ -1,16 +1,22 @@
 package com.blackduck.integration.detectable.detectables.cargo.parse;

-import java.util.Optional;
+import java.util.*;


Another wildcard import statement.

cpottsbd · 2025-05-13T19:06:16Z

src/main/java/com/blackduck/integration/detect/configuration/DetectProperties.java

+            .setInfo("Cargo Dependency Types Excluded", DetectPropertyFromVersion.VERSION_10_5_0)
+            .setHelp(
+                "A comma-separated list of dependency types that will be excluded.",
+                "The Cargo CLI Detector uses cargo tree flags to exclude the specified types, while the Cargo Lockfile Detector filters dependencies by reading Cargo.toml. For example, passing DEV will skip [dev-dependencies] from detection."


I might be inclined to pass the full parameter entry for the example?
i.e. "For example, passing DETECT_CARGO_DEPENDENCY_TYPES_EXCLUDED = DEV will skip [dev-dependencies] from detection."

Thanks. Updated as per the review.

…toml

…IDETECT-4326

.../main/java/com/blackduck/integration/detectable/detectables/cargo/parse/CargoTomlParser.java

…allback

andrian-sevastyanov · 2025-05-20T19:06:23Z

For lock file detectable, have you tested a scenario like this:

normal components: componentA
dev components: componentB
transitives: componentA dependends on componentB

I believe when excluding dev dependencies in such a case both componentA and componentB should be present in the result; however, in the case of lock file detectable we will exclude componentB.

Please correct me if I'm wrong.

andrian-sevastyanov · 2025-05-20T20:31:41Z

.../main/java/com/blackduck/integration/detectable/detectables/cargo/parse/CargoTomlParser.java

+                .allMatch(dependencyTypeFilter::shouldExclude);
+
+            if (shouldBeExcluded) {
+                dependenciesToExclude.put(nameVersion.getName(), nameVersion.getVersion());


This dependenciesToExclude map allows for a maximum of one version per dependency name; however, it looks like it's possible to specify incompatible versions of the same dependency if they are different types. Example excerpt from Cargo.toml:

[dev-dependencies] syn = "2.0.1" [build-dependencies] syn = "1"

results in the following tree:

[build-dependencies] └── syn v1.0.109 ├── proc-macro2 v1.0.95 │ └── unicode-ident v1.0.18 ├── quote v1.0.40 │ └── proc-macro2 v1.0.95 (*) └── unicode-ident v1.0.18 [dev-dependencies] └── syn v2.0.101 ├── proc-macro2 v1.0.95 (*) ├── quote v1.0.40 (*) └── unicode-ident v1.0.18

and Cargo.lock has:

... [[package]] name = "syn" version = "1.0.109" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "72b64191b275b66ffe2469e8af2c1cfe3bafa67b529ead792a6d0160888b4237" dependencies = [ "proc-macro2", "quote", "unicode-ident", ] [[package]] name = "syn" version = "2.0.101" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8ce2b7fc941b3a24138a0a7cf8e858bfc6a992e7978a068a5c760deb0ed43caf" dependencies = [ "proc-macro2", "quote", "unicode-ident", ] ...

Using a map of type Map<NameVersion, String> instead of Map<String, String> might solve this problem, as NameVersion key might be unique for a Cargo.toml file.

andrian-sevastyanov · 2025-05-20T20:39:59Z

I have an idea that could help resolve the special cases we have been discussing. For Cargo.lock, when excluding dependencies we could try something like this:

Go through sections of Cargo.toml to determine which sections should be included
For all included sections, figure out what the root dependencies are
Use the lock file to figure out the transitive dependencies for root dependencies from step 2
The final result is then combination of step 2 and 3.

I may have missed something, but I think it's worth considering at the very least.

andrian-sevastyanov · 2025-05-20T21:07:13Z

...ble/src/main/java/com/blackduck/integration/detectable/detectables/cargo/CargoExtractor.java

            .map(cargoLockPackageDataTransformer::transform)
            .collect(Collectors.toList());

        DependencyGraph graph = cargoLockPackageTransformer.transformToGraph(packages);

        Optional<NameVersion> projectNameVersion = Optional.empty();
        if (cargoTomlFile != null) {
-            String cargoTomlContents = FileUtils.readFileToString(cargoTomlFile, StandardCharsets.UTF_8);


Previously we were checking whether the file is not null before reading it. Now, we don't. FileUtils.readFileToString throws an exception.
We should continue to check whether the file is there before reading it.

Also, I think something like this should happen:

if cargoTomlFile == null and isDependencyExclusionEnabled(): failExtraction() // because we can't reliably determine dependency types; also, we might want to do this in the `extractable()` method of Detectable

andrian-sevastyanov · 2025-05-20T21:57:14Z

...lackduck/integration/detectable/detectables/cargo/transform/CargoLockPackageTransformer.java

+                    ExternalId childExternalId = externalIdFactory.createNameVersionExternalId(Forge.CRATES, childName, childVersion);
+
+                    graph.setDependencyInfo(childId, childName, childVersion, childExternalId);
+                    graph.setDependencyAsAlias(childId, LazyId.fromName(childName));


What are aliases used for? I've seen this before but not sure what the end effect is.

andrian-sevastyanov · 2025-05-20T21:58:00Z

.../src/main/java/com/blackduck/integration/detectable/detectables/cargo/CargoCliExtractor.java

+        exclusionMap.put(CargoDependencyType.NORMAL, "no-normal");
+        exclusionMap.put(CargoDependencyType.BUILD, "no-build");
+        exclusionMap.put(CargoDependencyType.DEV, "no-dev");
+        exclusionMap.put(CargoDependencyType.PROC_MACRO, "no-proc-macro");


Could you describe situations in which PROC_MACRO exclusion is useful? And I guess it's not applicable to lock file extractions?

…IDETECT-4326

zahidblackduck · 2025-05-22T12:56:39Z

I have an idea that could help resolve the special cases we have been discussing. For Cargo.lock, when excluding dependencies we could try something like this:

Go through sections of Cargo.toml to determine which sections should be included

For all included sections, figure out what the root dependencies are

Use the lock file to figure out the transitive dependencies for root dependencies from step 2

The final result is then combination of step 2 and 3.

I may have missed something, but I think it's worth considering at the very least.

Interesting idea. Further discussion might help on the implementation strategy.

zahidblackduck added 9 commits April 23, 2025 11:55

property to exclude dev dependencies in cargo cli detector add

10883f5

cargo cli extractor dev dependencies exclusion add

0354a8e

cargo lockfile detector dependency exclusion add

2830904

detect rule factory edit

1ac92d4

cargo version match util update

5344176

cargo version match util add

fba1188

cargo lockfile dev,build dependency exclusion add

f6e48d9

Merge conflict resolve with master

b66b9f7

cargo lock file dependency transformer refactor

37c4ed6

zahidblackduck marked this pull request as draft May 6, 2025 13:43

zahidblackduck self-assigned this May 6, 2025

zahidblackduck requested review from dterrybd, devmehtabd, shantyk and andrian-sevastyanov May 6, 2025 13:44

andrian-sevastyanov reviewed May 6, 2025

View reviewed changes

.../src/main/java/com/blackduck/integration/detectable/detectables/cargo/CargoCliExtractor.java Outdated Show resolved Hide resolved

andrian-sevastyanov requested a review from cpottsbd May 6, 2025 14:26

andrian-sevastyanov reviewed May 6, 2025

View reviewed changes

.../main/java/com/blackduck/integration/detectable/detectables/cargo/parse/CargoTomlParser.java Outdated Show resolved Hide resolved

andrian-sevastyanov reviewed May 6, 2025

View reviewed changes

...ble/src/main/java/com/blackduck/integration/detectable/detectables/cargo/CargoExtractor.java Outdated Show resolved Hide resolved

zahidblackduck added 2 commits May 7, 2025 15:29

wildcard imports cleared

9e1e4a4

method name refactor to express actual intent

95d1bac

devmehtabd reviewed May 7, 2025

View reviewed changes

doc description updated for new detect properties

1293fb9

cpottsbd reviewed May 13, 2025

View reviewed changes

zahidblackduck added 3 commits May 15, 2025 18:39

handle dependency exclusion if present in multiple sections of Cargo.…

1a0e93a

…toml

handle dependency exclusion if present in multiple sections of Cargo.…

0d8f6f2

…toml

properties doc updated with an example usage

d2a8a6c

zahidblackduck requested review from cpottsbd, andrian-sevastyanov and devmehtabd May 15, 2025 12:57

Merge remote-tracking branch 'origin/master' into dev/zahidblackduck/…

5eb9b16

…IDETECT-4326

cpottsbd approved these changes May 15, 2025

View reviewed changes

andrian-sevastyanov reviewed May 15, 2025

View reviewed changes

.../main/java/com/blackduck/integration/detectable/detectables/cargo/parse/CargoTomlParser.java Outdated Show resolved Hide resolved

refactor dependency exclusion to use NameVersion without precedence f…

d2be399

…allback

zahidblackduck requested a review from andrian-sevastyanov May 16, 2025 12:46

zahidblackduck added 2 commits May 16, 2025 18:49

remove code comment and empty line

83ebdf1

refactor condition to check actual dependency exclusion filter

ae1ee8e

devmehtabd approved these changes May 20, 2025

View reviewed changes

andrian-sevastyanov reviewed May 20, 2025

View reviewed changes

zahidblackduck added 2 commits May 22, 2025 14:14

Merge remote-tracking branch 'origin/master' into dev/zahidblackduck/…

383b99e

…IDETECT-4326

parse dependencies to exclude map key change to NameVersion

204b05f

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Detect property to exclude Cargo dependencies #1421

Detect property to exclude Cargo dependencies #1421

zahidblackduck commented May 6, 2025

andrian-sevastyanov May 6, 2025

devmehtabd May 7, 2025

zahidblackduck May 8, 2025 •

edited

Loading

devmehtabd May 8, 2025

zahidblackduck May 15, 2025

devmehtabd May 19, 2025

zahidblackduck May 19, 2025

devmehtabd May 7, 2025

zahidblackduck May 8, 2025

devmehtabd May 7, 2025

cpottsbd May 13, 2025

zahidblackduck May 15, 2025

andrian-sevastyanov commented May 20, 2025

andrian-sevastyanov May 20, 2025 •

edited

Loading

zahidblackduck May 22, 2025

andrian-sevastyanov commented May 20, 2025

andrian-sevastyanov May 20, 2025 •

edited

Loading

andrian-sevastyanov May 20, 2025

andrian-sevastyanov May 20, 2025

zahidblackduck commented May 22, 2025

Detect property to exclude Cargo dependencies #1421

Are you sure you want to change the base?

Detect property to exclude Cargo dependencies #1421

Conversation

zahidblackduck commented May 6, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zahidblackduck May 8, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrian-sevastyanov commented May 20, 2025

andrian-sevastyanov May 20, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrian-sevastyanov commented May 20, 2025

andrian-sevastyanov May 20, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zahidblackduck commented May 22, 2025

zahidblackduck May 8, 2025 •

edited

Loading

andrian-sevastyanov May 20, 2025 •

edited

Loading

andrian-sevastyanov May 20, 2025 •

edited

Loading